Method for estimating the motion between two digital images with management of mesh overturning and corresponding coding method

ABSTRACT

The invention relates to a method for estimating the motion between two digital images and to the use of this method for coding images. The aim of the method is to eliminate the disturbances caused by the mesh being overturned during the motion estimation process. According to the invention, the method consists of carrying out a first motion estimation in order to identify the faulty mesh, i.e. the mesh that is overturning or is overlapping after the displacement vectors have been applied; then carrying out a second motion estimation, excluding at least said faulty mesh, in order to minimize prediction error on the remaining mesh of the model.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention concerns a method for estimating the motionbetween two digital images and to the use of this method for codingimages.

The motion between two images, I₁ and I₂, is generally defined in theform of a field of motion associated with either of the images I₁ and I₂and constituted by motion vectors each related to a point of the imageconcerned. The motion vector is a two dimensional vector representativeof the position difference between the pixel of the image I₁ and theassociated pixel of the image I₂ relating to the same physical point ofthe filmed scene.

The invention can be applied in all fields of imagery requiring ananalysis of the movements or disparities between two images. This methodcan also form part of applying on follow up objects in videos (for thepurposes in particular of creating and annotating contents by theobjects in a standardisation context MPEG4 and 7).

So as to obtain a motion field, it is known to break down the image intofinite elements. These finite elements, which can be triangles orquadrangles, are determined by a mesh whose nodes correspond to thevertices of the finite elements. A motion vector is calculated for eachmesh node. Then, via an interpolation function, it is possible to deducefrom this a motion vector for each point of the image. The motion fieldis thus determined by the retained model of finite elements whichdefines the mesh used to partition the image into finite elements andthe interpolation function able to calculate the motion vector at anypoint of the image.

The mesh used can be regular or irregular and needs to be sufficientlydense so as to model as best as possible the motion between the twoimages but without requiring an excessive number of calculations or datato be transmitted.

The calculation of the motion vectors of the nodes of the mesh can becarried out according to various methods. First of all, there arematching methods consisting of testing a discrete set of possible valuesof motion vectors for each node of the mesh and of retaining the bestvectors according to a given criterion. A second method known as thetransform method consists of using the properties of the Fouriertransform and its extensions so as to convert the motion into a phasedisplacement in the transformed space. Finally, there is a third methodknown as the differential method which is able to determine the motionvectors by optimising a mathematical criterion (for example byoptimising a quadratic error between the image and its value predictedby the motion field). This method is the most used one for the motionestimation with modelising by finite elements. A traditionaldifferential method for optimising motion vectors is the Gauss-Newtonmethod. The present application concerns more particularly the motionestimation method family using a model of finite elements and adifferential method by optimising a prediction error for determining thefield of motion.

Although widely prevalent, this type of method does have one majordrawback. Under the effect of the field of the motion vectors of thenodes of the mesh, the initial mesh on the start image is transformedinto a new mesh on the other image. Then mesh overturnings may occur. Infact, the ductile mesh used to partition the image defines a continuousrepresentation of a field of motion, whereas the real motion of thefilmed scene is naturally discontinuous. Various elements of the scenecan appear or disappear from one image to another, which then generatediscontinuities. This is expressed at the level of the meshing by apulling or an overturning or an overlapping of the meshes of the model.A mesh overturning and overlapping example is illustrated by the FIGS.1A and 1B. FIG. 1A shows a ductile triangular meshing before applyingmotion vectors to the nodes of this meshing, and FIG. 1B shows the samemeshing after applying said motion vectors. Four nodes with thereferences A, B, C and D are indicated on these figures. These nodesconstitute the vertices of two triangles ABC and BCD. The triangle ABCis hatched on the two figures. It can be observed that, after applyingthe motion vectors (FIG. 1B), the triangle ABC has overturned and allthe triangles whose vertex is the node A cover the triangle BCD, thusdestroying the continuity property any meshing needs to check. Thisoverlapping of meshes then brings about a visual deterioration. Thus,one aim of the invention is to eliminate this visual deterioration.

2. Description of the Related Art

At the present time, there are two techniques for resolving these meshoverturning problems: one technique known as the “post-processing”technique and the technique consisting of establishing non-overturningconstraints when estimating the motion.

The “post-processing” technique can be implemented according to twopossible scenarios: the first scenario corresponding to an a posterioricorrection consists of applying as such the calculated motion vectors,of detecting the faulty motion vectors, and then of correcting theirvalue so as to inhibit the overturnings; the second scenario consists ofproceeding in an iterative way by applying at each iteration a portionof the expected motion to the nodes so that there is no overturning andby thus looping until the process has converged. However, given the factthat the post-processing methods act once the motion estimation isachieved, the result is sub-optimum because the motion vectors arecorrected independently of their global contribution in minimising theprediction error.

Another solution consists of optimising the motion field by integratingnon-overturning constraints of the triangles when estimating the motionvectors. The motion estimation is modified by adding to the predictionquadratic error an augmented Lagrangian for correcting the deformationof the triangles when the latter approach the zero area triangle. Thistechnique is described in the article entitled “Limitation of trianglesoverlapping in mesh-based motion estimation using augmented Lagrangian”,International Conference Image Processing, Chicago, October 1998. Thislatter technique would effectively make it possible to determine theoptimum solution to the problem if the motion field were continuous,which is not the case.

OBJECTS AND SUMMARY OF THE INVENTION

The solution adopted by the invention to resolve the problem ofoverturnings of meshes consists of making a first motion estimation soas to detect the faulty meshes, that is overturning or overlappingmeshes after applying motion vectors and of carrying out a second motionestimation by excluding at least said faulty meshes so as to minimisethe prediction error on the other meshes of the model.

Thus, the invention concerns a method for estimating the motion betweentwo digital images I₁ and I₂ with luminance Y₁ and Y₂, and intended togenerate for each point of co-ordinates x, y of the image I₂ a motionvector {right arrow over (d)}(x, y)=(d_(x), d_(y)) so as to form animage Î₂ from the image I₁ with luminance Ŷ₂(x, y)=Y₁(x+d_(x), y+d_(y))which is an approximation of the image I₂, comprising the followingsteps:

-   (a)—defining an initial model of finite elements comprising a mesh    whose nodes are points of the image I₁, a motion vector at each node    of said mesh, and an interpolation formula so as to calculate the    value of the motion vector of each point of the image I₁ on the    basis of the values of the motion vectors of the nodes of the mesh    to which it belongs,-   (b)—optimising globally the values of all the motion vectors of the    model according to a differential method,-   (c)—applying said motion vectors to said mesh of the initial model    so as to generate a moved mesh representative of the image Î₂,-   (d)—determining faulty areas in said moved mesh, said faulty areas    being determined so that they contain at least one or several meshes    overturning or overlapping after applying the motion vectors, all    said meshes of the moved meshwork overturning or overlapping after    applying the motion vectors being contained in said faulty areas,-   (e)—creating a set S including all the nodes of the moved mesh    contained in the faulty areas,-   (f)—taking up again the initial model defined in step (a),    reinitialising the values of all the motion vectors of said initial    model and re-optimising the values of the motion vectors of said    model according to step (b) by excluding from the model the motion    vectors of the nodes of said set S.

This double optimisation of the motion vectors makes it possible todetermine the optimal motion vectors for the continuous area (the areaallowing a bijection between the image I₁ and the image I₂) of the meshand of eliminating the disturbances brought by the motion vectors of thefaulty areas.

As a variant, it is possible to re-optimise the motion vectors of step(f) by taking up again, not the initial model, but a model correspondingto the model obtained at the end of an iteration of the optimisationcarried out in step (b) for which the optimised motion vectors do notgenerate any overturning of the meshes and so as to reduce theprocessing time of step (f).

It is also possible to provide a model of finite elements comprising ahierarchical mesh. In this case, the invention concerns a method forestimating the motion between two digital images, I₁ and I₂, withluminance Y₁ and Y₂, and intended to generate for each point ofcoordinates x, y of the image I₂ a motion vector {right arrow over(d)}(x, y)=(d_(x), d_(y)) so as to form an image Î₂ from the image I₁with luminance Ŷ₂(x, y)=Y₁(x+d_(x), y+d_(y)) which is an approximationof the image I₂, comprising the following steps:

-   (a)—defining an initial model of finite elements comprising a    hierarchical mesh whose nodes are points of the image I₁, said    hierarchical mesh comprising N+1 mesh levels (0 . . . N) and each    node of said mesh belonging to at least one mesh level, a motion    vector at each node of said hierarchical mesh, and an interpolation    formula so as to calculate the value of the motion vector of each    point of the image I₁ on the basis of the values of the motion    vectors of the nodes of the mesh to which it belongs,-   (b)—optimising globally, mesh level by mesh level, the values of all    the motion vectors of the model according to a differential method    by starting from the coarsest mesh level (level 0) and depending at    each level on the optimised values of the lower mesh level,-   (c)—applying said motion vectors to said hierarchical mesh of the    initial model so as to generate a moved mesh representative of the    image Î₂,-   (d)—determining the faulty areas at each mesh level of the moved    mesh, said faulty areas being determined so that they each contain    at least one or several overturning or overlapping meshes after    applying the motion vectors, all said meshes of said hierarchical    meshwork overturning or overlapping after applying the motion    vectors being contained in said faulty areas, the faulty areas of    the mesh of mesh level i including at least the faulty areas of the    mesh of mesh level i−1;-   (e)—creating a set S_(i) for each mesh level i, said set S_(i)    including all the nodes of the mesh level i of the moved mesh    contained in the faulty areas,-   (f)—taking up again the initial model defined in step (a),    reinitialising the values of all motion vectors of said initial    model and re-optimising the values of the motion vectors of said    model according to step (b) by excluding at each mesh level i of the    model the motion vectors of the nodes of the corresponding set    S_(i).

The implementation of this method to a sequence of video images makes itpossible to manage the disappearances of objects (overturnings ofmeshes) between consecutive images of the sequence by taking I₂consecutive to I₁. The appearances of objects between consecutivesequence images can also be managed by taking I₁ consecutive to I₂.

Finally, the invention also concerns a method for coding a sequence ofdigital images aimed at producing a binary train representative of saidsequence of images, characterised in that it comprises the followingsteps:

-   -   carrying out a motion estimation for each pair of consecutive        images of the sequence by implementing the previously mentioned        motion estimation method, the motion estimation being front or        rear, and    -   introducing, into the binary train, data for describing the        first image of said sequence, data concerning the structure of        the mesh used for the motion estimation step, the vector        displacements obtained at the end of said motion estimation step        and faulty data for describing the faulty areas determined        during the motion estimation step.

The invention also concerns a method for coding a sequence of digitalimages aimed at producing a binary train representative of said sequenceof images, said method being characterised in that it comprises thefollowing steps:

-   -   carrying out for each pair of consecutive images of the sequence        a front motion estimation and a rear motion estimation by        implementing for each of the motion estimations the motion        estimation method of the invention, and    -   introducing, into the binary train, data for describing the        first image of said sequence of images, data concerning the        structure of the mesh used for the motion estimation step,        motion vectors which for each image are a linear combination of        the motion vectors obtained at the end of the two preceding        motion estimation steps, and faulty data to represent the faulty        areas determined during the two preceding motion estimation        steps.

Advantageously, the binary train further comprises a bit indicatingwhether or not the motion estimation step of each of the images hasdetected faulty areas in the mesh.

BRIEF DESCRIPTION OF THE DRAWINGS

Other characteristics and advantages of the invention shall appear froma reading of the following detailed description with reference to theaccompanying drawings on which:

FIG. 1A, already described, represents a mesh on the image I₁ and themotion vectors associated with the nodes of this mesh for generating theimage I₂;

FIG. 1B, already described, shows the mesh of FIG. 1A after applying themotion vectors;

FIG. 2 represents a first embodiment of the method of the invention;

FIG. 3 shows an overturning area and a faulty area in the mesh of FIG.1A, and

FIG. 4 represents a second embodiment of the method of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Let us consider two digital images I₁ and I₂ belonging to a given videosequence and with respective luminance Y₁ and Y₂. The method of theinvention consists of carrying out a first motion estimation so as togenerate at each point P of coordinates (x, y) of the image I₂ a motionvector {right arrow over (d)}(x, y)=(d_(x), d_(y)). This vector isdefined as being the vector making it possible to construct from theimage I₁ an image Î₂ with luminance Ŷ₂(x, y)=Y₁(x+d_(x), y+d_(y)) whichis an approximation of I₂. The motions are thus defined from the imageI₁ to the image I₂. A point of coordinates (x, y) in the image I₁ hasthe coordinates (x+d_(x), y+d_(y)) in the image I₂.

In the rest of the description, a front motion estimation between theimages I₁ and I₂ is such that, when the image I₂ is consecutive to theimage I₁ in the sequence of images, the motion is estimated in thedirection I₁ towards I₂. For a rear motion estimation, the motion isestimated in the direction I₂ towards I₁.

The sought-after motion field is defined by a model of finite elements.In the continuation of the disclosure, the finite elements are regardedas triangles without imposing any limitation of the scope of the presentapplication to this form of finite elements. The model of finiteelements thus comprises a triangular mesh, motion vectors defined at thenodes of the mesh, said nodes corresponding to the vertices of thetriangles, and an interpolation formula for calculating the motionvector of the points inside the triangles.

The interpolation formula used to calculate the field of motion at anypoint of the domain of the image I₂ is the following:

-   -   if the point P of coordinates (x, y) in the image I₂ is        considered as belonging to the triangle e with vertices P_(i),        P_(j) and P_(k) with respective coordinates (x_(i), y_(i)),        (x_(j), y_(j)) and (x_(k), y_(k)), its motion vector is equal        to:        ${\overset{\rightarrow}{d}\left( {x,y} \right)} = {\sum\limits_{{1 = i},j,k}{{\psi_{1}^{e}\left( {x,y} \right)} \cdot {\overset{\rightarrow}{d}\left( {x_{1},y_{1}} \right)}}}$    -   where ψ₁^(e)        represents a base function associated with the triangle e.

In the case of an affine interpolation, the ψ_(l) ^(e)(x, y) representthe barycentric coordinates of the point P in the triangle e withvertices P_(l), P_(j), P_(k). These functions are defined by thefollowing equation: $\left\{ {{\begin{matrix}{\left\{ \begin{matrix}{{\psi_{1}^{e}\left( {x,y} \right)} = {\alpha_{1} + {\beta_{1}x} + {\gamma_{1}y}}} & {\left( {x,y} \right) \in e} \\{{\sum\limits_{{1 = i},j,k}{\psi_{1}^{e}\left( {x,y} \right)}} = 1} & {{{et}\quad\alpha_{1}},\beta_{1},{\gamma_{1} \in \Re}}\end{matrix} \right.} \\\begin{matrix}{{\psi_{1}^{e}\left( {x,y} \right)} = 0} & {{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}{\left( {x,y} \right) \notin e}}\end{matrix}\end{matrix}{so}},{{\psi_{i}^{e}\left( {x,y} \right)} = \frac{{x_{j}y_{k}} - {x_{k}y_{j}} + {\left( {y_{j} - y_{k}} \right)x} + {\left( {x_{k} - x_{j}} \right)y}}{{x_{j}y_{k}} - {x_{k}y_{j}} + {x_{k}y_{i}} - {x_{i}y_{k}} + {x_{i}y_{j}} - {x_{j}y_{k}}}}} \right.$

The affine functions ψ_(j)^(e)(x, y)  and  ψ_(k)^(e)(x, y)are deduced from the function ψ_(i)^(e)(x, y)by circularly permuting the indices i, j, k. It is also possible to usemore open-ended models of finite elements, the functions ψ then beingable to be extended to polynomials of degree n≧2 but the interpolationformula of the motion vectors then introduces first order, second orderderivatives and so on . . . A miscellaneous choice of models of finiteelements is proposed in the work “Handbook of Numerical Analysis” by P.G. Ciarlet and J. L Lions, Volume 2, pp 59-99, Ed. North Holland.

According to a first embodiment shown on FIG. 2, the motion estimationmethod relates to a model of finite elements including anon-hierarchical mesh.

According to a first step 10, an initial model of finite elements isdefined by selecting points of the image I₁ according to an initialtriangular mesh. This mesh can be defined in some ways, for exampleaccording to the requirements of the application or to prior knowledgeor to the motion already calculated between two preceding images of thesame video sequence. A zero value motion vector is then associated witheach node of the mesh. The interpolation formula defined previously isalso a data element of the initial model.

According to a second step 20, the value of the motion vectors of themodel are optimised according to a differential method, for example theGauss-Newton method or its Marquardt extension described in the thesisentitled “Représentation et codage de séquences vidéo par maillages 2Ddeformables” by Patrick Lechat, page 162, IRISA, Rennes, October 1999.This method consists of looking for all the nodal motion vectors whichminimise the prediction error between the images Î₂ and I₂. Thisprediction error is defined by:$E = {\sum\limits_{{({x,y})} \in D}{{DFD}^{2}\left( {x,y} \right)}}$

-   -   with DFD(x, y)=Y₂(x, y)−Y₁(x+d_(x), y+d_(y))    -   where D is the domain of the images I₁ and I₂

Advantageously, a local refining of the mesh via the division of thetriangles is then carried out when the prediction error E on certaintriangles of the mesh between the image Î₂ and the image I₂ is too high.In order to achieve this, the difference E is calculated between theimage Î₂ and the image I₂ on each triangle of the mesh and this triangleis divided if this difference is greater than a threshold value. Thenthe value of the motion vectors is re-optimised on the basis of this newmesh by repeating the preceding optimisation step. Thus, this sequenceof steps (optimisation of the motion vectors, calculation of E for eachtriangle, division of triangles) is repeated until a stop criterion issatisfied. This stop criterion may for example be a predetermined numberof triangles to be reached at the end of the local refining step. It isalso possible to stop the method when the differences E of all thetriangles are less than a threshold difference.

At the end of this first motion estimation, a model is obtained as shownon FIG. 1A with a given mesh and motion vectors for each node of thismesh. After applying the motion vectors to the nodes of the mesh (FIG.1B), certain triangles of the mesh overturn and others overlap. Thisapplication step is referenced 30 on the diagram of FIG. 2. In theexample of FIG. 1B, the triangle ABC overturns and the triangles fromwhich a vertex is the node A partly cover the triangle BCD.

According to the invention, the overturning areas are determined in themesh of the model in a step 40. These areas are made up of triangleswhich, after applying the motion vectors, either overturn or overlapother triangles. The mesh of FIG. 1A comprises an overturning area whichis shown on FIG. 3 and which includes all the triangles from which avertex is the node A. This area is delimited by the hexagon BCEFGH.

According to the invention, a set S is then created including all thenodes belonging to the overturning areas of the mesh (step with thereference 50 on FIG. 2). In the example of FIG. 3, the set S includesthe nodes A, B, C, E, F, G and H.

Advantageously, the set S is enlarged with the peripheral nodes of theoverturning area so as to avoid any risk of overturning of theperipheral triangles in the overturning area during a subsequentprocessing of the mesh (for example a new optimisation of the motionvectors). Then an area including the overturning area, known as a faultyarea, is defined. For example, let us consider the k-disk of theoverturning area in question. A faulty area representing the 2-disk ofthe hexagon BCEFGH is shown on FIG. 3. This faulty area is delimited bythe polygon P shown by the thick lines on the figure.

During the next step referenced 60 on FIG. 2, all the motion vectors ofthe initial model are then re-initialised, that is all the motionvectors of the nodes of the mesh of FIG. 1A, and the values of thesemotion vectors are re-optimised by excluding from the model the motionvectors of the nodes of the set S and this by reapplying theGauss-Newton method.

The motion vectors obtained at the end of the method are optimal for thecontinuous area of the mesh because the disturbance generated by thefaulty areas has been eliminated.

As a variant, it is possible to carry out re-optimisation of the motionvectors of the step (f) by taking up again, not the initial model, but amodel corresponding to the model obtained at the end of an iteration ofthe optimisation carried out in the step (b) for which the motionvectors do not generate any overturning of meshes. This therefore makesit possible to reduce the time for processing the step (f).

The motion of the nodes of the faulty areas of the image I₂ is notestimated with respect to the image I₁ since these areas correspond to aportion of the image I₂ which is not present in the image I₁.

Furthermore, it may be that the absence of certain nodes in the meshcreates degeneracies that generate problems of matrix conditioning atthe time of optimising the motion vectors. Also, so as to avoid thesematrix conditioning problems and accelerate the convergence of theGauss-Newton method, it is possible to apply the second optimisationstep to all the nodes of the mesh by firstly assuming that thedifference of luminance between the images I₁ and I₂ for the faultyareas is zero.

As a variant, it is also possible to correct the position of the nodesof the faulty areas of the mesh so as to limit the risks of degeneraciesduring optimisations of the motion vectors to follow (especially whenestimating the motion of the other images of the sequence of imagesincluding the images I₁ and I₂). This local correction of the mesh canbe carried out for example according to the barycenter method describedin the thesis entitled “Représentation et codage de séquences vidéo parmaillages 2D déformables” by Patrick Lechat, page 174, IRISA, Rennes,October 1999.

According to a second embodiment shown on FIG. 4, the model definedduring a first step 110 of the motion estimation method includes ahierarchical mesh which is made up of several nested meshes eachcorresponding to one mesh level. The mesh levels are numbered from 0 toN. The mesh level 0 corresponds to the coarser mesh and the mesh level Ncorresponds to the denser mesh. The level mesh i is obtained bysubdividing the meshes of the meshwork of level i−1. Thus, all the nodesof the mesh level i belong to the mesh level I+1. The motion vectorsassociated with the nodes of the hierarchical mesh are initially zeroand the interpolation function is the same as for the first embodiment.

According to a step referenced 120, the value of the motion vectors ofthe nodes of the model is then optimised by applying the Gauss-Newtonmethod to each mesh level by starting by the mesh level 0. Theoptimisation of the motion vectors is carried out mesh level by meshlevel, the values of the motion vectors of the nodes common to the meshlevels i and i+1 obtained at the end of optimisation at the level ibeing taken as initial values for these nodes during optimisation of thevectors at the level i+1. The motion vectors of the new nodes at themesh level i+1 are initialised by values obtained via the interpolationof the values of the motion vectors of the nodes of the level i. Then,in a step referenced 130, the calculated motion vectors are applied tothe hierarchical mesh of the model which discloses the faulty areas whenthe latter exist.

In this embodiment, the determination of the overturning or faulty areas(step 140) is carried out at each mesh level by starting with thecoarser mesh level (level 0). The faulty areas of the mesh at the meshlevel i include necessarily the faulty areas of the mesh at the leveli−1. Thus, the finest mesh level (level N) includes the greatest numberof faulty areas and the faulty areas the most extended. It would bepossible to envisage only carrying out a determination of theoverturning or faulty areas on the finest mesh, but it is essential totake account of the fact that certain faulty areas can disappear whenthe mesh becomes finer. These temporary faulty areas also disturboptimisation of the motion vectors and thus need to be identified.

Once said faulty areas are identified, it is then possible to nextcreate, for each mesh level i, a set S₁ containing the nodes of thefaulty zones of the mesh at the level i (step 150). It is to be notedthat, for any i ε[0 . . . N], S_(i)⊂S_(i+1).

Then, the initial model is taken up again; the value of the motionvectors of the nodes of the hierarchical mesh is re-initialised and thevalue of the motion vectors of the model is re-optimised by excludingfrom the model the motion vectors of the nodes of the faulty areas byapplying the Gauss-Newton method to each mesh level (step 160). Thisre-optimisation is first of all conducted on the motion vectors of themesh nodes of level 0 by excluding the nodes of the set S_(o), then onthe motion vectors of the nodes of the mesh of level 1 by excluding thenodes of the set S₁, and so on up to the level N.

So as to improve the position of the nodes of the moved mesh associatedwith the image Î₂, it is possible to estimate the motion between theimage Î₂ and the image I₂. In order to do this, a mesh is taken which iscomposed of the moved mesh for the continuous area of the image I₂ andof the initial mesh (that is the mesh of I₁) for the faulty areas. Themesh for the faulty areas can be corrected according to the barycentermethod. Then a motion estimation is made between the images Î₂ and I₂with this mesh and thus small value motion vectors are obtained (becauseÎ₂ is an approximation of I₂). By applying these motion vectors to thenodes of the moved mesh of the image I₂, a new moved mesh is obtainedwith the position of its nodes having improved.

In the two embodiments referred earlier, the first and second motionestimations of the method are carried out in the same direction: theyare either front to detect the disappearances of objects, or rear todetect the appearances of objects.

Again, in a more elaborate embodiment, it is possible to carry out afirst motion estimation which is a front motion estimation so as todetect the faulty areas corresponding to disappearances of objects, asecond motion estimation which is a rear motion estimation so as todetect the faulty areas corresponding to appearances of objects, andfinally a last motion estimation, either front or rear, by excludingfrom the estimation the faulty areas detected by the precedingestimations so as to exclude from the differential calculation all themeshes which could introduce errors.

The invention also concerns a method for coding fixed or animated imageswith flow reduction with a view to transmit or store them. Such a methodgenerates a binary train representative of the sequence of coded images.This coding method comprises a motion estimation implementing the methoddescribed previously. This motion estimation is carried out on each pairof consecutive images (I₁, I₂) of the sequence. This motion estimationcan be either a front motion estimation (the motion between I₁ and I₂ isthen calculated) or a rear motion estimation (the motion between I₂ andI₁ is then calculated). Of course, the motion of the first image of thesequence is not estimated. The data which is introduced into the binarytrain is then: data for describing the first image, data concerning thestructure of the start mesh used for estimating motion and the motionvectors of each image issued from the motion estimation. The datarelating to the first image is the structural data of a mesh used todescribe the first image and data concerning the luminance, chrominanceand positions of the nodes of this mesh. The mesh used for describingthe first image of the sequence can in fact differ from that used forestimating motion and may in particular comprise more mesh levels. As avariant, it is possible to carry out a front motion estimation and arear motion estimation according to the motion estimation method of theinvention and then introduce into the binary train, for each image,motion vectors which would be a linear combination of the motion vectorsfrom the two motion estimations.

Before being introduced into the binary train, the data of the firstimage of the sequence is coded differentially. In addition, all the dataconcerning luminance, chrominance and position to be introduced into thebinary train are quantified and compressed before being introduced intothe binary train so as to limit the number of possible values and reducethe amount of data to be transmitted or stored.

According to the invention, it is also possible to introduce into thebinary train for each image whose motion has been estimated a detectionbit indicating whether or not the motion estimation of this image hasgenerated overturnings of meshes. If this is the case, a specific valueis then reserved for the motion vectors of the nodes of the detectedoverturning areas. For example, the value 0 is allocated to the motionvectors of the overturning areas and the value of the other motionvectors is increased by one unit. The motion vectors with the value 0are used for detecting overturning areas in reception. The size k of thedisk corresponding to the faulty area is then introduced into the binarytrain so as to determine the nodes belonging to this area. It is alsopossible to allocate the value 0 to all the nodes of the faulty areas.It then becomes pointless to transmit the size of the k-disk.

Data for describing the faulty areas of each image is then introducedinto the binary train after the motion vectors. This data is for exampleobtained via an approximation by finite elements or by applying adiscrete cosine transform or a wavelet transform on the portion of theimage corresponding to these faulty areas.

As a variant, the data for describing the faulty areas introduced intothe binary train are the optimised values of the luminance andchrominance of the nodes of the set S, the optimisation of these valuesconsisting of modifying these values so that they represent as best aspossible these faulty areas. If the meshing is hierarchical, thisoptimisation can be effected in two different ways:

-   either the optimisation is carried out for all the faulty areas of    the image in question on all the nodes of the set S_(N) (the finest    mesh); this optimisation can then be carried out by using a    hierarchical approach (optimisation of the values of the nodes of    S₀, then values of the nodes of S₁ by being based on those of the    nodes of S₀, . . . up to S_(N)) or by directly optimising the values    of the nodes of S_(N);-   or each faulty area of the image in question is processed    individually and optimisation is then carried out on the nodes of    the set S_(j) belonging to the faulty area in question with j≧1,    being the mesh level where the faulty area in question has appeared;    similarly, this optimisation can be effected by following a    hierarchical approach or by directly optimising the values of the    nodes of the set S_(j).

According to the hierarchical approach, it is possible to startoptimisation of the values at a mesh level m>0 instead of starting atthe level 0.

The optimisation of the luminance and chrominance values can be effectedby a least mean square method which consists of minimising a criterionE′ defined on the domain of the faulty area Ω. This criterion is definedby the following expression:$E^{\prime} = {\int_{\Omega}{{{{I\left( {x,y} \right)} - {\sum\limits_{n = 1}^{M}{{\psi_{P_{n}}\left( {x,y} \right)} \cdot {v\left( P_{n} \right)}}}}}^{2}{\mathbb{d}x}{\mathbb{d}y}}}$where

-   -   P_(n) is a node with index number n of the mesh,    -   M is the total number of nodes of the set S or S_(i) in        question,    -   I(x, y) represents the luminance chrominance respectively) value        of the pixel with coordinates (x, y),    -   ψP_(n) is the interpolation function associated with the vertex        P_(n),    -   v(P_(n)) is the luminance (chrominance respectively) optimised        value associated with the vertex P_(n).

The operation for minimising this criterion is given in detail in thethesis entitled “Représentation et codage de séquences vidéo parmaillages 2D déformables” by Patrick Lechat, page 106, IRISA, Rennes,October 1999.

In addition, on receipt, the decoding of the binary train produced bythis coding method consists of:

-   -   decoding all the data included in the binary train,    -   determining if the detection bit indicates the presence of        faulty areas in the mesh during the motion estimation step of        the images of the coding method,    -   if this is the case, marking the faulty areas via the zero value        motion vectors,    -   decreasing by one unit the value of the non-zero value motion        vectors, and    -   recreating the sequence of images corresponding to the binary        train.

1. Method for estimating the motion between two digital images I₁ and I₂with luminance Y₁ and Y₂ and intended to generate for each point ofcoordinates x, y of the image I₂ a motion vector {right arrow over(d)}(x, y)=(d_(x), d_(y)) so as to form an image Î₂ from the image I₁with luminance Ŷ₂(x, y)=Y₁(x+d_(x), y+d_(y)) which is an approximationof the image I₂ and comprising the following steps: (a)—defining aninitial model of finite elements comprising a mesh whose nodes arepoints of the image I₁, a motion vector at each node of said mesh, andan interpolation formula so as to calculate the value of the motionvector of each point of the image I₁ on the basis of the values of themotion vectors of the nodes of the mesh to which it belongs,(b)—optimising globally the values of all the motion vectors of themodel according to a differential method, (c)—applying said motionvectors to said mesh of the initial model so as to generate a moved meshrepresentative of the image Î₂, (d)—determining faulty areas in saidmoved mesh, said faulty areas being determined so that they contain atleast one or several meshes overturning or overlapping after applyingthe motion vectors, all said meshes of the moved meshwork overturning oroverlapping after applying the motion vectors being contained in saidfaulty areas, (e)—creating a set S including all the nodes of the movedmesh contained in the faulty areas, (f)—taking up again the initialmodel defined in step (a), reinitialising the values of all the motionvectors of said initial model and re-optimising the values of the motionvectors of said model according to step (b) by excluding from the modelthe motion vectors of the nodes of said set S.
 2. Method according toclaim 1, characterised in that, instead of taking up again the initialmodel for reoptimisation of the motion vectors in step (f), a model istaken up corresponding to the model obtained at the end of an iterationof the optimisation carried out in step (b) for which the optimisedmotion vectors do not generate any overturning of meshes so as to reducethe processing time of step (f).
 3. Method according to claim 1,characterised in that it further comprises the following steps to becarried out between steps (b) and (c): (b1)—calculating a difference Ebetween the image Î₂ and the image I₂ for each mesh of the model,(b2)—carrying out a finer mesh on a discrete fraction of all the meshesdetermined according to a criterion relating to the differences E andallocating a motion vector to each new mesh node, (b3)—marking the steps(b), (b1) and (b2) on the model obtained at the end of the precedingstep (b2) until a stop criterion is satisfied.
 4. Method according toclaim 3, characterised in that, so as to carry out a finer mesh on adiscrete fraction of all the meshes in step (b2), all the differences Ecalculated in step (b1) are compared with a threshold difference andthose meshes whose differences E are greater than a threshold differenceare subdivided into smaller meshes.
 5. Method according to claim 3,characterised in that said stop criterion is a predetermined number ofmeshes in the model to be attained at the end of step (b2).
 6. Methodaccording to claim 3, characterised in that said stop criterion of step(b3) is satisfied when the differences E of all meshes of the modelobtained at the end of the preceding step (b2) are less than a thresholddifference.
 7. Method according to claim 1, characterised in that eachfaulty area is a k-disk of an area including at least one overturning oroverlapping mesh after application of the motion vectors.
 8. Methodaccording to claim 3, characterised in that it further comprises a stepfor correcting the position of the nodes of the faulty areas so as toimprove the convergence of step (b) for estimating the motion of imageswith respect to the image I₂.
 9. Method according to claim 1,characterised in that the images I₁ and I₂ belong to a given sequence ofdigital images and in that the image I₂ is consecutive to I₁ in saidsequence of digital images.
 10. Method according to claim 1,characterised in that the images I₁ and I₂ belong to a given sequence ofdigital images and in that the image I₁ is consecutive to I₂ in saidsequence of digital images.
 11. Application of the motion estimationmethod according to claim 1 for coding a sequence of digital images. 12.Method for coding a sequence of digital images for producing a binarytrain representative of said sequence of images, characterised in thatit comprises the following steps: estimating a front motion estimationfor each pair of consecutive images of the sequence by implementing themotion estimation method according to one of claim 1, and introducinginto the binary train data for describing the first image of saidsequence of images, data concerning the structure of the mesh used forthe motion estimation step, the motion vectors obtained at the end ofthe motion estimation step of each image and faulty data forrepresenting the faulty areas determined during the motion estimationstep.
 13. Method for coding a sequence of digital images so as toproduce a binary train representative of said sequence of images,characterised in that it comprises the following steps: carrying our arear motion estimation for each pair of consecutive images of thesequence by implementing the motion estimation method according to claim1, and introducing into the binary train data for describing the firstimage of said sequence of images, data concerning the structure of themesh used for the motion estimation step, the motion vectors obtained atthe end of the motion estimation step of each image, and faulty data forrepresenting the faulty areas determined during the motion estimationstep.
 14. Method for coding a sequence of digital images for producing abinary train representative of said sequence of images, characterised inthat it comprises the following steps: carrying out a front and a rearmotion estimation for each pair of consecutive images of the sequence byimplementing the mentioned motion estimation method for each of themotion estimations according to claim 1, and introducing into the binarytrain data for describing the first image of said sequence, dataconcerning the structure of the mesh used for the motion estimationstep, motion vectors which are for each image a linear combination ofthe motion vectors obtained at the end of the two preceding motionestimation steps, and faulty data for representing the faulty areasdetermined during the two preceding motion estimation steps.
 15. Methodaccording to claim 12, characterised in that the binary train furthercomprises a detection bit indicating whether the motion estimation stepof each of the images has determined faulty areas in the mesh. 16.Method according to claim 12, characterised in that the value of themotion vectors obtained at the end of said motion estimation step ismodified before introducing said motion vectors into the binary train.17. Method according to claim 16, characterised in that the value zerois allocated to the motion vectors of the nodes of the faulty areas andin that the value of the other motion vectors of the model is increasedby one.
 18. Method for decoding a binary train representative of asequence of images and produced by the coding method according to claim17, characterised in that it comprises the following steps: decoding allthe data included in said binary train, determining whether thedetection bit indicates the presence of faulty areas of the mesh duringthe motion estimation step of the images of the coding method, if thisis the case, marking said faulty areas via the zero value motionvectors, decreasing by one unit the value of the positive value motionvectors, and recreating said sequence of images.
 19. Method according toclaim 12, characterised in that the faulty data introduced into thebinary train is obtained by applying a discrete cosine transform to thecorresponding portion of the digital image.
 20. Method according toclaim 12, characterised in that the faulty data introduced into thebinary train is obtained by applying a wavelet transform to thecorresponding portion of the digital image.
 21. Method according toclaim 12, characterised in that the faulty data introduced into thebinary train is obtained by applying a finite elements approximationmethod to the corresponding portion of the digital image.
 22. Methodaccording to claim 12, characterised in that the faulty data introducedinto the binary train is luminance, chrominance and possibly positionvalues of the nodes of the set S.
 23. Method according to claim 22,characterised in that the luminance and chrominance values of the nodesof the set S are optimised before being introduced into the binarytrain.
 24. Method according to claim 23, characterised in that theposition values of the nodes of the set S are optimised before beingintroduced into the binary train.
 25. Method according to claim 12,characterised in that the faulty data introduced into the binary trainis luminance, chrominance and position values of the nodes of the setS_(N) relating to the highest mesh level (N).
 26. Method according toclaim 25, characterised in that the luminance and chrominance values ofthe nodes of the set S_(N) are optimised before being introduced intothe binary train.
 27. Method according to claim 26, characterised inthat the position values of the nodes of the set S_(N) are optimisedbefore being introduced into the binary train.
 28. Method according toclaim 26, characterised in that the optimisation of the luminance andchrominance values of the nodes of the set S_(N) relating to the highestmesh level (N) is effected according to a hierarchical approach by firstof all optimising the luminance and chrominance values of the nodes ofthe set S₀ corresponding to the coarsest mesh level, then the luminanceand chrominance values of the nodes of the intermediate sets S_(i) bybeing based on the luminance and chrominance values of the nodes of thelower mesh level S_(i−1), until reaching the highest mesh level (N). 29.Method according to claim 28, characterised in that optimisationaccording to a hierarchical approach of the luminance and chrominancevalues of the set S_(N) is effected by starting by optimising theluminance and chrominance values of the nodes of the set S_(m) with m>0.30. Method according to claim 12, characterised in that the faulty dataintroduced into the binary train are luminance, chrominance and positionvalues of the nodes of the set S_(j) with j≧1, being the mesh levelwhere the faulty area in question has appeared.
 31. Method according toclaim 30, characterised in that the luminance and chrominance values ofthe nodes of the set S_(j) are optimised before being introduced intothe binary train.
 32. Method according to claim 31, characterised inthat the position values of the nodes of the set S_(j) are optimisedbefore being introduced into the binary train.
 33. Method according toclaim 31, characterised in that the optimisation of the luminance andchrominance values of the nodes of the set S_(j) is effected accordingto a hierarchical approach by first of all optimising the luminance andchrominance values of the nodes of the set S₀ corresponding to thecoarsest mesh level, then the luminance and chrominance values of thenodes of the intermediate sets S_(i) by being based on the luminance andchrominance values of the nodes of the lower mesh level S_(i−1) untilreaching the mesh level j.
 34. Method according to claim 33,characterised in that the optimisation according to a hierarchicalapproach of the luminance and chrominance values of the set S_(j) iseffected by starting by optimising the luminance and chrominance valuesof the nodes of the set S_(m) with 0<m<j.